
 <!DOCTYPE HTML>
<html>
<head><meta name="generator" content="Hexo 3.9.0">
  <meta charset="UTF-8">
  
    <title>容器界新势力cri-o | Zong&#39;s blog</title>
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=3, minimum-scale=1">
    
    <meta name="author" content="Zong">
    
    <meta name="description" content="一 介绍CRI-O - OCI-based implementation of Kubernetes Container Runtime Interface这是cri-o的github标题，符合OCI基准实现的Kubernetes容器运行时接口，从这个标题很容易看出，这是一个专门服务k8s的容器实现">
    
    
    
    
    
    <link rel="icon" href="/img/favicon.ico">
    
    
    <link rel="apple-touch-icon" href="/img/pacman.jpg">
    <link rel="apple-touch-icon-precomposed" href="/img/pacman.jpg">
    
    <link rel="stylesheet" href="/css/style.css">
</head>
</html>
  <body>
    <header>
      <div>
		
			<div id="imglogo">
				<a href="/"><img src="/img/logo.svg" alt="Zong&#39;s blog" title="Zong&#39;s blog"/></a>
			</div>
			
			<div id="textlogo">
				<h1 class="site-name"><a href="/" title="Zong&#39;s blog">Zong&#39;s blog</a></h1>
				<h2 class="blog-motto">日常积累，技术分享</h2>
			</div>
			<div class="navbar"><a class="navbutton navmobile" href="#" title="Menu">
			</a></div>
			<nav class="animated">
				<ul>
					<ul>
					 
						<li><a href="/">Home</a></li>
					
						<li><a href="/archives">Archives</a></li>
					
						<li><a href="/categories/运维">运维</a></li>
					
						<li><a href="/categories/容器架构">容器架构</a></li>
					
					<li>
					
					<form class="search" action="//baidu.com/s" method="get" accept-charset="utf-8">
						<label>Search</label>
						<input type="text" id="search" name="wd" autocomplete="off" maxlength="20" placeholder="Search" />
                        <input name=tn type=hidden value="bds">
                        <input name=cl type=hidden value="3">
                        <input name=ct type=hidden value="2097152">
						<input type="hidden" name="si" value="www.lstop.pub">
					</form>
					
					</li>
				</ul>
			</nav>			
</div>

    </header>
    <div id="container">
      <div id="main" class="post" itemscope itemprop="blogPost">
	<article itemprop="articleBody"> 
		<header class="article-info clearfix">
  <h1 itemprop="name">
    
      <a href="/2020/01/08/容器界新势力cri-o/" title="容器界新势力cri-o" itemprop="url">容器界新势力cri-o</a>
  </h1>
  <p class="article-author">By
    
      <a href="http://www.lstop.pub" title="Zong">Zong</a>
    </p>
  <p class="article-time">
    <time datetime="2020-01-08T06:29:46.000Z" itemprop="datePublished">2020-01-08</time>
    
  </p>
</header>

	<div class="article-content">
		
		
		<div id="toc" class="toc-article">
			<strong class="toc-title">Contents</strong>
		<ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#一-介绍"><span class="toc-number">1.</span> <span class="toc-text">一 介绍</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#二-对比docker"><span class="toc-number">2.</span> <span class="toc-text">二 对比docker</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#1-比docker简洁"><span class="toc-number">2.1.</span> <span class="toc-text">1 比docker简洁</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#2-命令精简"><span class="toc-number">2.2.</span> <span class="toc-text">2 命令精简</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#三-安装配置"><span class="toc-number">3.</span> <span class="toc-text">三 安装配置</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#cri-o"><span class="toc-number">3.1.</span> <span class="toc-text">cri-o</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#cri-tools"><span class="toc-number">3.2.</span> <span class="toc-text">cri-tools</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#四-整合kubelet"><span class="toc-number">4.</span> <span class="toc-text">四 整合kubelet</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#五-可能不足的地方"><span class="toc-number">5.</span> <span class="toc-text">五 可能不足的地方</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#版本问题"><span class="toc-number">5.1.</span> <span class="toc-text">版本问题</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#稳定性"><span class="toc-number">5.2.</span> <span class="toc-text">稳定性</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#六-出现的问题"><span class="toc-number">6.</span> <span class="toc-text">六 出现的问题</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#获取不到容器监控指标"><span class="toc-number">6.1.</span> <span class="toc-text">获取不到容器监控指标</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#不能创建容器"><span class="toc-number">6.2.</span> <span class="toc-text">不能创建容器</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#kubelet重复报Failed-to-create-existing-container"><span class="toc-number">6.3.</span> <span class="toc-text">kubelet重复报Failed to create existing container</span></a></li></ol></li></ol>
		</div>
		
		<h1 id="一-介绍"><a href="#一-介绍" class="headerlink" title="一 介绍"></a>一 介绍</h1><p>CRI-O - OCI-based implementation of Kubernetes Container Runtime Interface<br>这是<a href="https://github.com/cri-o/cri-o" target="_blank" rel="noopener">cri-o</a>的github标题，符合OCI基准实现的Kubernetes容器运行时接口，从这个标题很容易看出，这是一个专门服务k8s的容器实现。从他的版本发布规律也很容易得出这个结论，因为cri-o的版本号和k8s版本基本一致。<br>这里面有2个概念比较容易混淆</p>
<ul>
<li>CRI Container Runtime Interface这是k8s提出的一个概念，在容器界除了最出名的docker和本文介绍的cri-o，还有很多不同的容器实现，例如contrainerd, frakti，rkt等，这些容器实现各有特色，只要支持CRI就可以被k8s支持</li>
<li>OCI Open Container Initiative这是开放容器标准，也叫容器runtime，简单来说只要符合这个标准运行的就是容器，例如runC，Kata，gVisor这些runtime创建出来的都是OCI标准容器容器标准，也叫容器runtime，简单来说只要符合这个标准运行的，就是容器，例如runC，Kata，gVisor这些runtime创建出来的都是OCI标准容器</li>
</ul>
<blockquote>
<p><a href="https://aleiwu.com/post/cncf-runtime-landscape/" target="_blank" rel="noopener">白话 Kubernetes Runtime</a> 这篇文章写得挺有趣，可以读一下加深了解<br><a href="http://weekly.dockerone.com/article/8891" target="_blank" rel="noopener">为Kubernetes选择合适的容器运行时</a> 这篇更详细一点</p>
</blockquote>
<h1 id="二-对比docker"><a href="#二-对比docker" class="headerlink" title="二 对比docker"></a>二 对比docker</h1><h2 id="1-比docker简洁"><a href="#1-比docker简洁" class="headerlink" title="1 比docker简洁"></a>1 比docker简洁</h2><p>cri-o对比docker的一大特点就是直接，调用链条短。熟悉docker架构就知道，docker是传统的守护进程架构，docker把命令发送到docker daemon，docker daemon调用contrainerd进程，创建containerd-shim进程调用runC启动容器。整个架构显得繁琐，因为docker本身也有一套集群编排架构swarm，所以很多规则和调用放在k8s架构下显得重。<br>而cri-o就直接调用runC创建容器，而且容器并不会下挂到crio进程下面，天生没有docker重启容器的尴尬。<br>下面这个图直观描述了调用链</p>
<p><img src="https://i.imgur.com/nUAYZiq.png" alt></p>
<h2 id="2-命令精简"><a href="#2-命令精简" class="headerlink" title="2 命令精简"></a>2 命令精简</h2><p>cri-o本身实现的命令没有docker全，其中一个考虑是够用就好，例如没有push，build这些生产环境不需要的命令，所以他本身适合做单纯的容器环境。</p>
<h1 id="三-安装配置"><a href="#三-安装配置" class="headerlink" title="三 安装配置"></a>三 安装配置</h1><h2 id="cri-o"><a href="#cri-o" class="headerlink" title="cri-o"></a>cri-o</h2><p>安装挺简单，ubuntu可以用apt</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">sudo apt-add-repository ppa:projectatomic/ppa</span><br><span class="line">sudo apt-get update -qq</span><br><span class="line">sudo apt-get install cri-o-[REQUIRED VERSION]</span><br></pre></td></tr></table></figure>

<p>版本可以和k8s环境一致，也可以高一个版本</p>
<p>配置文件修改一下 /etc/crio/crio.conf</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br><span class="line">156</span><br><span class="line">157</span><br><span class="line">158</span><br><span class="line">159</span><br><span class="line">160</span><br><span class="line">161</span><br><span class="line">162</span><br><span class="line">163</span><br><span class="line">164</span><br><span class="line">165</span><br><span class="line">166</span><br><span class="line">167</span><br><span class="line">168</span><br><span class="line">169</span><br><span class="line">170</span><br><span class="line">171</span><br><span class="line">172</span><br><span class="line">173</span><br><span class="line">174</span><br><span class="line">175</span><br><span class="line">176</span><br><span class="line">177</span><br><span class="line">178</span><br><span class="line">179</span><br><span class="line">180</span><br><span class="line">181</span><br><span class="line">182</span><br><span class="line">183</span><br><span class="line">184</span><br><span class="line">185</span><br><span class="line">186</span><br><span class="line">187</span><br><span class="line">188</span><br><span class="line">189</span><br><span class="line">190</span><br><span class="line">191</span><br><span class="line">192</span><br><span class="line">193</span><br><span class="line">194</span><br><span class="line">195</span><br><span class="line">196</span><br><span class="line">197</span><br><span class="line">198</span><br><span class="line">199</span><br><span class="line">200</span><br><span class="line">201</span><br><span class="line">202</span><br><span class="line">203</span><br><span class="line">204</span><br><span class="line">205</span><br><span class="line">206</span><br><span class="line">207</span><br><span class="line">208</span><br><span class="line">209</span><br><span class="line">210</span><br><span class="line">211</span><br><span class="line">212</span><br><span class="line">213</span><br><span class="line">214</span><br><span class="line">215</span><br><span class="line">216</span><br><span class="line">217</span><br><span class="line">218</span><br><span class="line">219</span><br><span class="line">220</span><br><span class="line">221</span><br><span class="line">222</span><br><span class="line">223</span><br><span class="line">224</span><br><span class="line">225</span><br><span class="line">226</span><br><span class="line">227</span><br><span class="line">228</span><br><span class="line">229</span><br><span class="line">230</span><br><span class="line">231</span><br><span class="line">232</span><br><span class="line">233</span><br><span class="line">234</span><br><span class="line">235</span><br><span class="line">236</span><br><span class="line">237</span><br><span class="line">238</span><br><span class="line">239</span><br><span class="line">240</span><br><span class="line">241</span><br><span class="line">242</span><br><span class="line">243</span><br><span class="line">244</span><br><span class="line">245</span><br><span class="line">246</span><br><span class="line">247</span><br><span class="line">248</span><br><span class="line">249</span><br><span class="line">250</span><br><span class="line">251</span><br><span class="line">252</span><br><span class="line">253</span><br><span class="line">254</span><br><span class="line">255</span><br><span class="line">256</span><br><span class="line">257</span><br><span class="line">258</span><br><span class="line">259</span><br><span class="line">260</span><br><span class="line">261</span><br><span class="line">262</span><br><span class="line">263</span><br><span class="line">264</span><br><span class="line">265</span><br><span class="line">266</span><br><span class="line">267</span><br><span class="line">268</span><br><span class="line">269</span><br><span class="line">270</span><br><span class="line">271</span><br><span class="line">272</span><br><span class="line">273</span><br><span class="line">274</span><br><span class="line">275</span><br><span class="line">276</span><br></pre></td><td class="code"><pre><span class="line"># The CRI-O configuration file specifies all of the available configuration</span><br><span class="line"># options and command-line flags for the crio(8) OCI Kubernetes Container Runtime</span><br><span class="line"># daemon, but in a TOML format that can be more easily modified and versioned.</span><br><span class="line">#</span><br><span class="line"># Please refer to crio.conf(5) for details of all configuration options.</span><br><span class="line"></span><br><span class="line"># CRI-O reads its storage defaults from the containers-storage.conf(5) file</span><br><span class="line"># located at /etc/containers/storage.conf. Modify this storage configuration if</span><br><span class="line"># you want to change the system&apos;s defaults. If you want to modify storage just</span><br><span class="line"># for CRI-O, you can change the storage configuration options here.</span><br><span class="line">[crio]</span><br><span class="line"></span><br><span class="line"># Path to the &quot;root directory&quot;. CRI-O stores all of its data, including</span><br><span class="line"># containers images, in this directory.</span><br><span class="line">#root = &quot;/home/lsm5/.local/share/containers/storage&quot;</span><br><span class="line"></span><br><span class="line"># Path to the &quot;run directory&quot;. CRI-O stores all of its state in this directory.</span><br><span class="line">#runroot = &quot;/tmp/1000&quot;</span><br><span class="line"></span><br><span class="line"># Storage driver used to manage the storage of images and containers. Please</span><br><span class="line"># refer to containers-storage.conf(5) to see all available storage drivers.</span><br><span class="line">#storage_driver = &quot;vfs&quot;</span><br><span class="line"></span><br><span class="line"># List to pass options to the storage driver. Please refer to</span><br><span class="line"># containers-storage.conf(5) to see all available storage options.</span><br><span class="line">#storage_option = [</span><br><span class="line">#]</span><br><span class="line"></span><br><span class="line"># If set to false, in-memory locking will be used instead of file-based locking.</span><br><span class="line">file_locking = false</span><br><span class="line"></span><br><span class="line"># Path to the lock file.</span><br><span class="line">file_locking_path = &quot;/run/crio.lock&quot;</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># The crio.api table contains settings for the kubelet/gRPC interface.</span><br><span class="line">[crio.api]</span><br><span class="line"></span><br><span class="line"># Path to AF_LOCAL socket on which CRI-O will listen.</span><br><span class="line">listen = &quot;/var/run/crio/crio.sock&quot;</span><br><span class="line"></span><br><span class="line"># IP address on which the stream server will listen.</span><br><span class="line">stream_address = &quot;127.0.0.1&quot;</span><br><span class="line"></span><br><span class="line"># The port on which the stream server will listen.</span><br><span class="line">stream_port = &quot;0&quot;</span><br><span class="line"></span><br><span class="line"># Enable encrypted TLS transport of the stream server.</span><br><span class="line">stream_enable_tls = false</span><br><span class="line"></span><br><span class="line"># Path to the x509 certificate file used to serve the encrypted stream. This</span><br><span class="line"># file can change, and CRI-O will automatically pick up the changes within 5</span><br><span class="line"># minutes.</span><br><span class="line">stream_tls_cert = &quot;&quot;</span><br><span class="line"></span><br><span class="line"># Path to the key file used to serve the encrypted stream. This file can</span><br><span class="line"># change, and CRI-O will automatically pick up the changes within 5 minutes.</span><br><span class="line">stream_tls_key = &quot;&quot;</span><br><span class="line"></span><br><span class="line"># Path to the x509 CA(s) file used to verify and authenticate client</span><br><span class="line"># communication with the encrypted stream. This file can change, and CRI-O will</span><br><span class="line"># automatically pick up the changes within 5 minutes.</span><br><span class="line">stream_tls_ca = &quot;&quot;</span><br><span class="line"></span><br><span class="line"># Maximum grpc send message size in bytes. If not set or &lt;=0, then CRI-O will default to 16 * 1024 * 1024.</span><br><span class="line">grpc_max_send_msg_size = 16777216</span><br><span class="line"></span><br><span class="line"># Maximum grpc receive message size. If not set or &lt;= 0, then CRI-O will default to 16 * 1024 * 1024.</span><br><span class="line">grpc_max_recv_msg_size = 16777216</span><br><span class="line"></span><br><span class="line"># The crio.runtime table contains settings pertaining to the OCI runtime used</span><br><span class="line"># and options for how to set up and manage the OCI runtime.</span><br><span class="line">[crio.runtime]</span><br><span class="line"></span><br><span class="line"># A list of ulimits to be set in containers by default, specified as</span><br><span class="line"># &quot;&lt;ulimit name&gt;=&lt;soft limit&gt;:&lt;hard limit&gt;&quot;, for example:</span><br><span class="line"># &quot;nofile=1024:2048&quot;</span><br><span class="line"># If nothing is set here, settings will be inherited from the CRI-O daemon</span><br><span class="line">default_ulimits = [</span><br><span class="line">  &quot;nofile=5120:5120&quot;</span><br><span class="line">]</span><br><span class="line"></span><br><span class="line"># default_runtime is the _name_ of the OCI runtime to be used as the default.</span><br><span class="line"># The name is matched against the runtimes map below.</span><br><span class="line">default_runtime = &quot;runc&quot;</span><br><span class="line"></span><br><span class="line"># If true, the runtime will not use pivot_root, but instead use MS_MOVE.</span><br><span class="line">no_pivot = false</span><br><span class="line"></span><br><span class="line"># Path to the conmon binary, used for monitoring the OCI runtime.</span><br><span class="line">conmon = &quot;/usr/bin/conmon&quot;</span><br><span class="line"></span><br><span class="line"># Environment variable list for the conmon process, used for passing necessary</span><br><span class="line"># environment variables to conmon or the runtime.</span><br><span class="line">conmon_env = [</span><br><span class="line">        &quot;PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin&quot;,</span><br><span class="line">]</span><br><span class="line"></span><br><span class="line"># If true, SELinux will be used for pod separation on the host.</span><br><span class="line">selinux = false</span><br><span class="line"></span><br><span class="line"># Path to the seccomp.json profile which is used as the default seccomp profile</span><br><span class="line"># for the runtime.</span><br><span class="line">seccomp_profile = &quot;/etc/crio/seccomp.json&quot;</span><br><span class="line"></span><br><span class="line"># Used to change the name of the default AppArmor profile of CRI-O. The default</span><br><span class="line"># profile name is &quot;crio-default-&quot; followed by the version string of CRI-O.</span><br><span class="line">apparmor_profile = &quot;crio-default&quot;</span><br><span class="line"></span><br><span class="line"># Cgroup management implementation used for the runtime.</span><br><span class="line">cgroup_manager = &quot;systemd&quot;</span><br><span class="line"></span><br><span class="line"># List of default capabilities for containers. If it is empty or commented out,</span><br><span class="line"># only the capabilities defined in the containers json file by the user/kube</span><br><span class="line"># will be added.</span><br><span class="line">default_capabilities = [</span><br><span class="line">        &quot;CHOWN&quot;, </span><br><span class="line">        &quot;DAC_OVERRIDE&quot;, </span><br><span class="line">        &quot;FSETID&quot;, </span><br><span class="line">        &quot;FOWNER&quot;, </span><br><span class="line">        &quot;NET_RAW&quot;, </span><br><span class="line">        &quot;SETGID&quot;, </span><br><span class="line">        &quot;SETUID&quot;, </span><br><span class="line">        &quot;SETPCAP&quot;, </span><br><span class="line">        &quot;NET_BIND_SERVICE&quot;, </span><br><span class="line">        &quot;SYS_CHROOT&quot;, </span><br><span class="line">        &quot;KILL&quot;, </span><br><span class="line">]</span><br><span class="line"></span><br><span class="line"># List of default sysctls. If it is empty or commented out, only the sysctls</span><br><span class="line"># defined in the container json file by the user/kube will be added.</span><br><span class="line">default_sysctls = [</span><br><span class="line">]</span><br><span class="line"></span><br><span class="line"># List of additional devices. specified as</span><br><span class="line"># &quot;&lt;device-on-host&gt;:&lt;device-on-container&gt;:&lt;permissions&gt;&quot;, for example: &quot;--device=/dev/sdc:/dev/xvdc:rwm&quot;.</span><br><span class="line">#If it is empty or commented out, only the devices</span><br><span class="line"># defined in the container json file by the user/kube will be added.</span><br><span class="line">additional_devices = [</span><br><span class="line">]</span><br><span class="line"></span><br><span class="line"># Path to OCI hooks directories for automatically executed hooks.</span><br><span class="line">hooks_dir = [</span><br><span class="line">]</span><br><span class="line"></span><br><span class="line"># List of default mounts for each container. **Deprecated:** this option will</span><br><span class="line"># be removed in future versions in favor of default_mounts_file.</span><br><span class="line">default_mounts = [</span><br><span class="line">]</span><br><span class="line"></span><br><span class="line"># Path to the file specifying the defaults mounts for each container. The</span><br><span class="line"># format of the config is /SRC:/DST, one mount per line. Notice that CRI-O reads</span><br><span class="line"># its default mounts from the following two files:</span><br><span class="line">#</span><br><span class="line">#   1) /etc/containers/mounts.conf (i.e., default_mounts_file): This is the</span><br><span class="line">#      override file, where users can either add in their own default mounts, or</span><br><span class="line">#      override the default mounts shipped with the package.</span><br><span class="line">#</span><br><span class="line">#   2) /usr/share/containers/mounts.conf: This is the default file read for</span><br><span class="line">#      mounts. If you want CRI-O to read from a different, specific mounts file,</span><br><span class="line">#      you can change the default_mounts_file. Note, if this is done, CRI-O will</span><br><span class="line">#      only add mounts it finds in this file.</span><br><span class="line">#</span><br><span class="line">#default_mounts_file = &quot;&quot;</span><br><span class="line"></span><br><span class="line"># Maximum number of processes allowed in a container.</span><br><span class="line">pids_limit = 1024</span><br><span class="line"></span><br><span class="line"># Maximum sized allowed for the container log file. Negative numbers indicate</span><br><span class="line"># that no size limit is imposed. If it is positive, it must be &gt;= 8192 to</span><br><span class="line"># match/exceed conmon&apos;s read buffer. The file is truncated and re-opened so the</span><br><span class="line"># limit is never exceeded.</span><br><span class="line">log_size_max = 536870912 </span><br><span class="line"></span><br><span class="line"># Whether container output should be logged to journald in addition to the kuberentes log file</span><br><span class="line">log_to_journald = false</span><br><span class="line"></span><br><span class="line"># Path to directory in which container exit files are written to by conmon.</span><br><span class="line">container_exits_dir = &quot;/var/run/crio/exits&quot;</span><br><span class="line"></span><br><span class="line"># Path to directory for container attach sockets.</span><br><span class="line">container_attach_socket_dir = &quot;/var/run/crio&quot;</span><br><span class="line"></span><br><span class="line"># If set to true, all containers will run in read-only mode.</span><br><span class="line">read_only = false</span><br><span class="line"></span><br><span class="line"># Changes the verbosity of the logs based on the level it is set to. Options</span><br><span class="line"># are fatal, panic, error, warn, info, and debug.</span><br><span class="line">log_level = &quot;error&quot;</span><br><span class="line"></span><br><span class="line"># The UID mappings for the user namespace of each container. A range is</span><br><span class="line"># specified in the form containerUID:HostUID:Size. Multiple ranges must be</span><br><span class="line"># separated by comma.</span><br><span class="line">uid_mappings = &quot;&quot;</span><br><span class="line"></span><br><span class="line"># The GID mappings for the user namespace of each container. A range is</span><br><span class="line"># specified in the form containerGID:HostGID:Size. Multiple ranges must be</span><br><span class="line"># separated by comma.</span><br><span class="line">gid_mappings = &quot;&quot;</span><br><span class="line"></span><br><span class="line"># The minimal amount of time in seconds to wait before issuing a timeout</span><br><span class="line"># regarding the proper termination of the container.</span><br><span class="line">ctr_stop_timeout = 0</span><br><span class="line"></span><br><span class="line">  # The &quot;crio.runtime.runtimes&quot; table defines a list of OCI compatible runtimes.</span><br><span class="line">  # The runtime to use is picked based on the runtime_handler provided by the CRI.</span><br><span class="line">  # If no runtime_handler is provided, the runtime will be picked based on the level</span><br><span class="line">  # of trust of the workload.</span><br><span class="line">  </span><br><span class="line">  [crio.runtime.runtimes.runc]</span><br><span class="line">  runtime_path = &quot;/usr/lib/cri-o-runc/sbin/runc&quot;</span><br><span class="line">  runtime_type = &quot;&quot;</span><br><span class="line">  </span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># The crio.image table contains settings pertaining to the management of OCI images.</span><br><span class="line">#</span><br><span class="line"># CRI-O reads its configured registries defaults from the system wide</span><br><span class="line"># containers-registries.conf(5) located in /etc/containers/registries.conf. If</span><br><span class="line"># you want to modify just CRI-O, you can change the registries configuration in</span><br><span class="line"># this file. Otherwise, leave insecure_registries and registries commented out to</span><br><span class="line"># use the system&apos;s defaults from /etc/containers/registries.conf.</span><br><span class="line">[crio.image]</span><br><span class="line"></span><br><span class="line"># Default transport for pulling images from a remote container storage.</span><br><span class="line">default_transport = &quot;docker://&quot;</span><br><span class="line"></span><br><span class="line"># The path to a file containing credentials necessary for pulling images from</span><br><span class="line"># secure registries. The file is similar to that of /var/lib/kubelet/config.json</span><br><span class="line">global_auth_file = &quot;&quot;</span><br><span class="line"></span><br><span class="line"># The image used to instantiate infra containers.</span><br><span class="line">pause_image = &quot;ccr.ccs.tencentyun.com/google_container/pause-amd64:3.1&quot;</span><br><span class="line"></span><br><span class="line"># The path to a file containing credentials specific for pulling the pause_image from</span><br><span class="line"># above. The file is similar to that of /var/lib/kubelet/config.json</span><br><span class="line">pause_image_auth_file = &quot;&quot;</span><br><span class="line"></span><br><span class="line"># The command to run to have a container stay in the paused state.</span><br><span class="line">pause_command = &quot;/pause&quot;</span><br><span class="line"></span><br><span class="line"># Path to the file which decides what sort of policy we use when deciding</span><br><span class="line"># whether or not to trust an image that we&apos;ve pulled. It is not recommended that</span><br><span class="line"># this option be used, as the default behavior of using the system-wide default</span><br><span class="line"># policy (i.e., /etc/containers/policy.json) is most often preferred. Please</span><br><span class="line"># refer to containers-policy.json(5) for more details.</span><br><span class="line">signature_policy = &quot;&quot;</span><br><span class="line"></span><br><span class="line"># Controls how image volumes are handled. The valid values are mkdir, bind and</span><br><span class="line"># ignore; the latter will ignore volumes entirely.</span><br><span class="line">image_volumes = &quot;mkdir&quot;</span><br><span class="line"></span><br><span class="line"># List of registries to be used when pulling an unqualified image (e.g.,</span><br><span class="line"># &quot;alpine:latest&quot;). By default, registries is set to &quot;docker.io&quot; for</span><br><span class="line"># compatibility reasons. Depending on your workload and usecase you may add more</span><br><span class="line"># registries (e.g., &quot;quay.io&quot;, &quot;registry.fedoraproject.org&quot;,</span><br><span class="line"># &quot;registry.opensuse.org&quot;, etc.).</span><br><span class="line">registries = [</span><br><span class="line">       &quot;quay.io&quot;,</span><br><span class="line">       &quot;docker.io&quot;</span><br><span class="line">]</span><br><span class="line"></span><br><span class="line">insecure_registries = [&quot;registry.bxr.cn&quot;]</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"># The crio.network table containers settings pertaining to the management of</span><br><span class="line"># CNI plugins.</span><br><span class="line">[crio.network]</span><br><span class="line"></span><br><span class="line"># Path to the directory where CNI configuration files are located.</span><br><span class="line">network_dir = &quot;/etc/cni/net.d/&quot;</span><br><span class="line"></span><br><span class="line"># Paths to directories where CNI plugin binaries are located.</span><br><span class="line">plugin_dirs = [</span><br><span class="line">        &quot;/opt/cni/bin&quot;,</span><br><span class="line">]</span><br></pre></td></tr></table></figure>

<ul>
<li>cgroup_manager 这个要和kubelet配置一致，推荐设置systemd</li>
<li>log_size_max 建议配一下，不要让log塞满磁盘</li>
<li>pause_image 换一个国内的吧</li>
<li>registries 取消掉注释，不然要写完整image路径</li>
<li>insecure_registries 内部registry</li>
</ul>
<p>配置systemd启动<br>/etc/systemd/system/crio.service </p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line">[Unit]</span><br><span class="line">Description=OCI-based implementation of Kubernetes Container Runtime Interface</span><br><span class="line">Documentation=https://github.com/kubernetes-incubator/cri-o</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">ExecStart=/usr/bin/crio --log-level info </span><br><span class="line">Restart=on-failure</span><br><span class="line">RestartSec=5</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br></pre></td></tr></table></figure>

<h2 id="cri-tools"><a href="#cri-tools" class="headerlink" title="cri-tools"></a>cri-tools</h2><p><a href="https://github.com/kubernetes-sigs/cri-tools" target="_blank" rel="noopener">cri-tools</a>是一个管理cri-o的命令工具，也适用其他CRI标准的容器。<br>下载对应版本的二进制包解压就能用，注意配置一下 /etc/crictl.yaml 里面的runtime-endpoint<br>该有的命令都有了，当然前面提到比docker命令少，有个特色是能管理pod维度</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">ubuntu@k8s-dev-node4:/etc/kubernetes$ sudo crictl pods</span><br><span class="line">POD ID              CREATED             STATE               NAME                                NAMESPACE           ATTEMPT</span><br><span class="line">dbdb04f7200a1       2 hours ago         Ready               metrics-server-549bdfcbf7-kvz56     kube-system         0</span><br><span class="line">8944933980db5       2 hours ago         Ready               coredns-9d458994c-bgxqm             kube-system         0</span><br><span class="line">0dd7b784b65a3       6 hours ago         Ready               calico-node-ssg6f                   kube-system         0</span><br></pre></td></tr></table></figure>

<h1 id="四-整合kubelet"><a href="#四-整合kubelet" class="headerlink" title="四 整合kubelet"></a>四 整合kubelet</h1><p>如果之前有docker，需要把运行的容器停止。在kubelet启动参数加上：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">--container-runtime=remote</span><br><span class="line">--container-runtime-endpoint=/var/run/crio/crio.sock</span><br><span class="line">--runtime-request-timeout=10m</span><br></pre></td></tr></table></figure>

<blockquote>
<p>网上包括官方doc都是–container-runtime-endpoint=unix:///var/run/crio/crio.sock，但是这样会引起一个问题，看第五章</p>
</blockquote>
<p>如果之前kubelet的systemd配置了docker前置条件，需要改一下</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">[Unit]</span><br><span class="line">Description=Kubernetes Kubelet</span><br><span class="line">Documentation=https://github.com/kubernetes/kubernetes</span><br><span class="line">After=crio.service</span><br><span class="line">Requires=crio.service</span><br><span class="line"></span><br><span class="line">[Service]</span><br><span class="line">EnvironmentFile=/etc/kubernetes/kubelet</span><br><span class="line">WorkingDirectory=/var/lib/kubelet</span><br><span class="line">ExecStart=/usr/bin/kubelet $KUBELET_ARGS</span><br><span class="line">Restart=on-failure</span><br><span class="line"></span><br><span class="line">[Install]</span><br><span class="line">WantedBy=multi-user.target</span><br></pre></td></tr></table></figure>

<p>然后重启kubelet</p>
<p>验证：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">ubuntu@k8s-dev-m1:~$ kubectl get no -o wide</span><br><span class="line">NAME                   STATUS                        ROLES      AGE     VERSION    INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                                        KERNEL-VERSION      CONTAINER-RUNTIME</span><br><span class="line">k8s-dev-node1          Ready                         k8s-node   22d     v1.13.5    10.22.0.2     &lt;none&gt;        Ubuntu 16.04.1 LTS                              4.4.0-91-generic    docker://18.3.1</span><br><span class="line">k8s-dev-node3.bxr.cn   Ready                         k8s-node   306d    v1.13.4    10.22.0.40    &lt;none&gt;        Container Linux by CoreOS 1745.5.0 (Rhyolite)   4.14.44-coreos-r1   docker://18.3.1</span><br><span class="line">k8s-dev-node4          Ready                         k8s-node   418d    v1.13.4    10.22.1.5     &lt;none&gt;        Ubuntu 16.04.1 LTS                              4.4.0-130-generic   cri-o://1.14.10</span><br><span class="line">k8s-dev-node5          Ready                         k8s-node   418d    v1.13.0    10.22.0.23    &lt;none&gt;        Ubuntu 16.04.1 LTS                              4.4.0-130-generic   docker://18.6.1</span><br></pre></td></tr></table></figure>

<p>可以看到k8s-dev-node4的CONTAINER-RUNTIME已经换成cri-o</p>
<h1 id="五-可能不足的地方"><a href="#五-可能不足的地方" class="headerlink" title="五 可能不足的地方"></a>五 可能不足的地方</h1><h2 id="版本问题"><a href="#版本问题" class="headerlink" title="版本问题"></a>版本问题</h2><p>crio版本和k8s版本的发布保持一致，官方也说使用版本最好和k8s保持1个小版差异，这样在升级k8s版本的时候可能要多考虑一下crio的版本，而k8s一般能兼容多个版本的docker</p>
<h2 id="稳定性"><a href="#稳定性" class="headerlink" title="稳定性"></a>稳定性</h2><p>稳定性方面好像还没经过大厂使用验证，也没有docker使用广泛，社区肯定不如docker活跃</p>
<h1 id="六-出现的问题"><a href="#六-出现的问题" class="headerlink" title="六 出现的问题"></a>六 出现的问题</h1><h2 id="获取不到容器监控指标"><a href="#获取不到容器监控指标" class="headerlink" title="获取不到容器监控指标"></a>获取不到容器监控指标</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">Jan  9 09:53:25 localhost kubelet[6482]: E0109 09:53:25.371839    6482 handler.go:306] HTTP InternalServerError serving /stats/summary: Internal Error: failed to get imageFs stats: failed to get imageFs info: no imagefs label for configured runtime</span><br><span class="line">Jan  9 09:53:45 localhost kubelet[6482]: E0109 09:53:45.569308    6482 kubelet.go:1304] Image garbage collection failed multiple times in a row: failed to get imageFs info: no imagefs label for configured runtime</span><br></pre></td></tr></table></figure>

<p>这个报错是kubelet的10255端口获取不到容器指标，测试：</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">ubuntu@k8s-dev-node4:~$ curl localhost:10255/stats/summary</span><br><span class="line">Internal Error: failed to get imageFs stats: failed to get imageFs info: no imagefs label for configured runtimeubuntu@k8s-dev-node4:~$</span><br></pre></td></tr></table></figure>

<p>参考github的issue <a href="https://github.com/kubernetes/kubernetes/issues/71712" target="_blank" rel="noopener">https://github.com/kubernetes/kubernetes/issues/71712</a><br>kubelet 1.13 配置runtime要去掉unix://</p>
<figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">--container-runtime-endpoint=/var/run/crio/crio.sock</span><br></pre></td></tr></table></figure>

<h2 id="不能创建容器"><a href="#不能创建容器" class="headerlink" title="不能创建容器"></a>不能创建容器</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">Jan  9 11:07:26 localhost kubelet[18240]: E0109 11:07:26.818320   18240 manager.go:1147] Failed to create existing container: /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cecfa7f_e89f_11e8_8d00_525400df7e25.slice/crio-0dd7b784b65a3b7fef40c0634b92b5663f49b53c0a9b176f674ab7517efa717f.scope: invalid character &apos;c&apos; looking for beginning of value</span><br></pre></td></tr></table></figure>

<p>重启cri-o后，kubelet一直报Failed to create existing container，观察集群pod和node节点pods状态正常，暂未发现实质影响</p>
<h2 id="kubelet重复报Failed-to-create-existing-container"><a href="#kubelet重复报Failed-to-create-existing-container" class="headerlink" title="kubelet重复报Failed to create existing container"></a>kubelet重复报Failed to create existing container</h2><figure class="highlight plain"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">E0224 18:17:35.083060   10955 manager.go:1086] Failed to create existing container: /kubepods/besteffort/pode26e7d50-4521-4962-a988-d646bb0fe953/crio-b3b80307234c60c4cc9ed419f9f402977abb253304ef3d6755dff85943f5dd5e: Error finding container b3b80307234c60c4cc9ed419f9f402977abb253304ef3d6755dff85943f5dd5e: Status 404 returned error &amp;&#123;%!s(*http.body=&amp;&#123;0xc001883fe0 &lt;nil&gt; &lt;nil&gt; false false &#123;0 0&#125; false false false &lt;nil&gt;&#125;) &#123;%!s(int32=0) %!s(uint32=0)&#125; %!s(bool=false) &lt;nil&gt; %!s(func(error) error=0x74f5f0) %!s(func() error=0x74f580)&#125;</span><br></pre></td></tr></table></figure>

<p>cgroup_manager设置systemd<br>在节点查看是否有NotReady的pods并删掉 sudo crictl rmp  podid</p>
  
	</div>
		<footer class="article-footer clearfix">

  <div class="article-tags">
  
  <span></span> <a href="/tags/容器/">容器</a>
  </div>


<div class="article-categories">
  <span></span>
  <a class="article-category-link" href="/categories/容器架构/">容器架构</a>
</div>



<div class="article-share" id="share">

  <div data-url="http://www.lstop.pub/2020/01/08/容器界新势力cri-o/" data-title="容器界新势力cri-o | Zong&#39;s blog" data-tsina="" class="share clearfix">
  </div>

</div>
</footer>   	       
	</article>
	
<nav class="article-nav clearfix">
 
 <div class="prev" >
 <a href="/2020/01/08/ubuntu18下的coredns踩坑/" title="ubuntu18下的coredns踩坑">
  <strong>PREVIOUS:</strong><br/>
  <span>
  ubuntu18下的coredns踩坑</span>
</a>
</div>


<div class="next">
<a href="/2019/12/31/calico-tunl-ip冲突/"  title="calico tunl ip冲突">
 <strong>NEXT:</strong><br/> 
 <span>calico tunl ip冲突
</span>
</a>
</div>

</nav>

	
</div>  
      <div class="openaside"><a class="navbutton" href="#" title="Show Sidebar"></a></div>

  <div id="toc" class="toc-aside">
  <strong class="toc-title">Contents</strong>
  <ol class="toc"><li class="toc-item toc-level-1"><a class="toc-link" href="#一-介绍"><span class="toc-number">1.</span> <span class="toc-text">一 介绍</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#二-对比docker"><span class="toc-number">2.</span> <span class="toc-text">二 对比docker</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#1-比docker简洁"><span class="toc-number">2.1.</span> <span class="toc-text">1 比docker简洁</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#2-命令精简"><span class="toc-number">2.2.</span> <span class="toc-text">2 命令精简</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#三-安装配置"><span class="toc-number">3.</span> <span class="toc-text">三 安装配置</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#cri-o"><span class="toc-number">3.1.</span> <span class="toc-text">cri-o</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#cri-tools"><span class="toc-number">3.2.</span> <span class="toc-text">cri-tools</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#四-整合kubelet"><span class="toc-number">4.</span> <span class="toc-text">四 整合kubelet</span></a></li><li class="toc-item toc-level-1"><a class="toc-link" href="#五-可能不足的地方"><span class="toc-number">5.</span> <span class="toc-text">五 可能不足的地方</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#版本问题"><span class="toc-number">5.1.</span> <span class="toc-text">版本问题</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#稳定性"><span class="toc-number">5.2.</span> <span class="toc-text">稳定性</span></a></li></ol></li><li class="toc-item toc-level-1"><a class="toc-link" href="#六-出现的问题"><span class="toc-number">6.</span> <span class="toc-text">六 出现的问题</span></a><ol class="toc-child"><li class="toc-item toc-level-2"><a class="toc-link" href="#获取不到容器监控指标"><span class="toc-number">6.1.</span> <span class="toc-text">获取不到容器监控指标</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#不能创建容器"><span class="toc-number">6.2.</span> <span class="toc-text">不能创建容器</span></a></li><li class="toc-item toc-level-2"><a class="toc-link" href="#kubelet重复报Failed-to-create-existing-container"><span class="toc-number">6.3.</span> <span class="toc-text">kubelet重复报Failed to create existing container</span></a></li></ol></li></ol>
  </div>

<div id="asidepart">
<div class="closeaside"><a class="closebutton" href="#" title="Hide Sidebar"></a></div>
<aside class="clearfix">

  
<div class="tagslist">
	<p class="asidetitle">Tags</p>
		<ul class="clearfix">
		
			<li><a href="/tags/Airtest/" title="Airtest">Airtest<sup>1</sup></a></li>
		
			<li><a href="/tags/DNS/" title="DNS">DNS<sup>1</sup></a></li>
		
			<li><a href="/tags/GitLab/" title="GitLab">GitLab<sup>1</sup></a></li>
		
			<li><a href="/tags/K8s/" title="K8s">K8s<sup>8</sup></a></li>
		
			<li><a href="/tags/Linux/" title="Linux">Linux<sup>1</sup></a></li>
		
			<li><a href="/tags/MongoDB/" title="MongoDB">MongoDB<sup>2</sup></a></li>
		
			<li><a href="/tags/OpenWrt/" title="OpenWrt">OpenWrt<sup>1</sup></a></li>
		
			<li><a href="/tags/Python/" title="Python">Python<sup>2</sup></a></li>
		
			<li><a href="/tags/RabbitMQ/" title="RabbitMQ">RabbitMQ<sup>1</sup></a></li>
		
			<li><a href="/tags/calico/" title="calico">calico<sup>1</sup></a></li>
		
			<li><a href="/tags/cdn/" title="cdn">cdn<sup>1</sup></a></li>
		
			<li><a href="/tags/docker/" title="docker">docker<sup>3</sup></a></li>
		
			<li><a href="/tags/docker-registry/" title="docker registry">docker registry<sup>1</sup></a></li>
		
			<li><a href="/tags/elasticsearch/" title="elasticsearch">elasticsearch<sup>3</sup></a></li>
		
			<li><a href="/tags/elk/" title="elk">elk<sup>3</sup></a></li>
		
			<li><a href="/tags/k8s/" title="k8s">k8s<sup>3</sup></a></li>
		
			<li><a href="/tags/kubernetes/" title="kubernetes">kubernetes<sup>1</sup></a></li>
		
			<li><a href="/tags/nginx/" title="nginx">nginx<sup>1</sup></a></li>
		
			<li><a href="/tags/python/" title="python">python<sup>1</sup></a></li>
		
			<li><a href="/tags/tomcat/" title="tomcat">tomcat<sup>1</sup></a></li>
		
		</ul>
</div>


  <div class="linkslist">
  <p class="asidetitle">Links</p>
    <ul>
      <li><a href="http://www.v2ex.com/?r=zong400" target="_blank" title="V2EX">V2EX</a></li>
      <li><a href="http://hexo.io" target="_blank" title="Hexo">Hexo</a></li>
	  <li><a href="https://promotion.aliyun.com/ntms/yunparter/invite.html?userCode=s0bh6uzq" target="_blank" title="阿里云">阿里云</a></li>
	  <li><a href="https://cloud.tencent.com/redirect.php?redirect=1014&cps_key=5bd9deb84d4d9f34b65fb934e12d03e3&from=console" target="_blank" title="腾讯云">腾讯云</a></li>
    </ul>
</div>


</aside>
</div>
    </div>
    <footer><div id="footer" >
	
	
	<div class="social-font" class="clearfix">
		
		
		
		
	</div>
		<p class="copyright">Hosted by <a href="https://pages.coding.me/" target="_blank" title="Coding Pages">Coding Pages</a></p>
		<p class="copyright">Powered by <a href="http://hexo.io" target="_blank" title="hexo">hexo</a> and Theme by <a href="https://github.com/wizicer/iceman" target="_blank" title="Iceman">Iceman</a> © 2020 
		
		<a href="http://www.lstop.pub" target="_blank" title="Zong">Zong</a>
		
		</p>
</div>
</footer>
    <script src="//cdn.staticfile.org/jquery/2.1.0/jquery.min.js"></script>
<script type="text/javascript">
$(document).ready(function(){ 
  $('.navbar').click(function(){
    $('header nav').toggleClass('shownav');
  });
  var myWidth = 0;
  function getSize(){
    if( typeof( window.innerWidth ) == 'number' ) {
      myWidth = window.innerWidth;
    } else if( document.documentElement && document.documentElement.clientWidth) {
      myWidth = document.documentElement.clientWidth;
    };
  };
  var m = $('#main'),
      a = $('#asidepart'),
      c = $('.closeaside'),
      o = $('.openaside');
  $(window).resize(function(){
    getSize(); 
    if (myWidth >= 1024) {
      $('header nav').removeClass('shownav');
    }else
    {
      m.removeClass('moveMain');
      a.css('display', 'block').removeClass('fadeOut');
      o.css('display', 'none');
      
      $('#toc.toc-aside').css('display', 'none');
        
    }
  });
  c.click(function(){
    a.addClass('fadeOut').css('display', 'none');
    o.css('display', 'block').addClass('fadeIn');
    m.addClass('moveMain');
  });
  o.click(function(){
    o.css('display', 'none').removeClass('beforeFadeIn');
    a.css('display', 'block').removeClass('fadeOut').addClass('fadeIn');      
    m.removeClass('moveMain');
  });
  $(window).scroll(function(){
    o.css("top",Math.max(80,260-$(this).scrollTop()));
  });
});
</script>

<script type="text/javascript">
$(document).ready(function(){ 
  var ai = $('.article-content>iframe'),
      ae = $('.article-content>embed'),
      t  = $('#toc'),
      h  = $('article h2')
      ah = $('article h2'),
      ta = $('#toc.toc-aside'),
      o  = $('.openaside'),
      c  = $('.closeaside');
  if(ai.length>0){
    ai.wrap('<div class="video-container" />');
  };
  if(ae.length>0){
   ae.wrap('<div class="video-container" />');
  };
  if(ah.length==0){
    t.css('display','none');
  }else{
    c.click(function(){
      ta.css('display', 'block').addClass('fadeIn');
    });
    o.click(function(){
      ta.css('display', 'none');
    });
    $(window).scroll(function(){
      ta.css("top",Math.max(140,320-$(this).scrollTop()));
    });
  };
});
</script>


<script type="text/javascript">
$(document).ready(function(){ 
  var $this = $('.share'),
      url = $this.attr('data-url'),
      encodedUrl = encodeURIComponent(url),
      title = $this.attr('data-title'),
      tsina = $this.attr('data-tsina');
  var html = [
  '<a href="#" class="overlay" id="qrcode"></a>',
  '<div class="qrcode clearfix"><span>扫描二维码分享到微信朋友圈</span><a class="qrclose" href="#share"></a><strong>Loading...Please wait</strong><img id="qrcode-pic" data-src="http://s.jiathis.com/qrcode.php?url=' + encodedUrl + '"/></div>',
  '<a href="#textlogo" class="article-back-to-top" title="Top"></a>',
  '<a href="https://www.facebook.com/sharer.php?u=' + encodedUrl + '" class="article-share-facebook" target="_blank" title="Facebook"></a>',
  '<a href="#qrcode" class="article-share-qrcode" title="QRcode"></a>',
  '<a href="https://twitter.com/intent/tweet?url=' + encodedUrl + '" class="article-share-twitter" target="_blank" title="Twitter"></a>',
  '<a href="http://service.weibo.com/share/share.php?title='+title+'&url='+encodedUrl +'&ralateUid='+ tsina +'&searchPic=true&style=number' +'" class="article-share-weibo" target="_blank" title="Weibo"></a>',
  '<span title="Share to"></span>'
  ].join('');
  $this.append(html);
  $('.article-share-qrcode').click(function(){
    var imgSrc = $('#qrcode-pic').attr('data-src');
    $('#qrcode-pic').attr('src', imgSrc);
    $('#qrcode-pic').load(function(){
        $('.qrcode strong').text(' ');
    });
  });
});     
</script>









  </body>
</html>

